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(54) Speaker adaptation based on eigenvolces 

(57) The recognizer tests input utterances using a 
confidence measure to select words of high recognition 
confidence for use in the adaptation process. Adapta- 
tion is performed rapidly using a priori knowledge of 
about the class of speakers who will be using the sys- 
tem. This a priori knowledge can be expressed using 
eignevoice basis vectors that capture information about 
the entire targeted user population. The dialogue sys- 



Speech 
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tern may also use the confidence measure to output a 
pronunciation example to the user, based on the confi- 
dence that the system has in the results of recognition, 
given the different possibilities that can be recognized. 
The dialogue system may also provide voiced prompts 
that teach the user how to correctly pronounce words. 
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Description 

Background and Summary of the Invention 

s [0001] The present invention relates generally to speech recognition systems and speech training systems. More 
particularly, the invention relates to a speech recognition apparatus having an adaptation system employing eigenvoice 
basis vectors to rapidly adapt the inrtfal speech model to that of the user. The system further employs a confidence 
measuring technique whereby the system automatically bases its adaptation upon utterances recognized with high 
confidence, while ignoring utterances recognized with low confidence. In this way, the system automatically adapts to 

70 the user quite rapidly, increasing the recognizer's chance of having good recognition performance, without adapting to 
incorrect pronunciations. The system is thus useful with difficult speakers, such as children or foreign speakers. 
[0002] Those who have used present day continuous speech recognition systems are acquainted with the time 
consuming and rigorous process by which the speech model of the recognizer is adapted to the individual user's 
speech. For most aduJt users who are already fluent in speaking the language, the adaptation process simply involves 

75 the discipline of providing sufficient ©camples of that user's speech so that the initially supplied speaker-independent 
speech model can be adapted into a speaker-dependent model for that speaker. The adaptation process can be super- 
vised, in which the user speaks words, phrases or sentences that are known in advance by the recognition system. 
Alternative/ the adaptation process can be unsupervised, in which casethe user speaks into the system without the 
system having a priori knowledge of the speech content. 

20 [0003] Adapting the speech recognition system to speech provided by children or foreign speakers who do not flu- 
ently speak the language is considerably more difficult.. Speech recognition systems have great difficulty correctly 
processing and recognizing the speech of children and foreign speakers, in part because the speech models of present 
day recognizers are trained upon a corpus of native-speaking adult speakers. There is simply very little data available 
for children and foreign speakers. 

25 [0004] In addition to poor recognition, speech recognizers have difficulty with children and foreign speakers 
because interaction with these types of users is very difficult. Children between the ages of four and seven generally 
have a difficult time concentrating on the task of training the recognizer. Children become distracted easily and cannot 
be relied upon to follow the adaptation procedures correctly. Indeed, this difficulty in obtaining speech data from children 
is one reason why the corpus of speech data from children is so small. 

so [0005] Foreign speakers present a similar problem. Although adult foreign speakers are able to concentrate on the 
adaptation task, like children, they may be unable to read training scripts used for adaptation, and they may mispro- 
nounce so many words that the adapted speech model will fail to properly recognize subsequent speech. 
[0006] The present invention addresses the foregoing problems by providing a speech recognition apparatus that 
will adapt the initial speech model using a highly effective and rapid adaptation system that will automatically assess 

35 the quality or accuracy of the user's pronunciation, using only the high confidence utterances for adaptation. The adap- 
tation system uses a priori knowledge about the class of speakers for which the application is intended to adapt to the 
user's voice with only a very limited amount of adaptation data. 

[0007] More specifically, the adaptation system is based on a speaker space representation of the class of speak- 
ers. A plurality of training speakers is used to generate speech models that are then dimensionally reduced to generate 

<c a set of basis vectors that define an eigenspace. During the adaptation process, speech units uttered by the user are 
used to train the adapted speech model, while the space spanned by the basis vectors is used to constrain the adapted 
speech model to lie within the eigenspace. As more fully described below, we have discovered that this eigenvoice tech- 
nique of encoding a priori knowledge about the target user population achieves remarkably rapid adaptation, even 
when very little adaptation data is provided. The system is able to begin performing adaptation almost as quickly as the 

as user begins speaking. Once the speaker has provided an utterance that the confidence measurement system admits 
as reliable, the speech model associated with that utterance may be placed or projected into the eigenspace. thereby 
establishing an adapted speech model that is constrained to the class of speakers for which the application is intended. 
[0008] The speech recognition apparatus has many uses and makes possible a number of interesting applications 
that have heretofore been difficult to achieve. One example is the computer-based teaching system that guides children 

so or foreign speakers in the correct pronunciation of new words within the language, in a system suitable for children, a 
simple supervised adaptation session can commence by prompting the child simply to state his or her name. The sys- 
tem may have a priori knowledge of the child's name by spelled-name keyboard entry. 

[0009] In a language teaching system the confidence measure can also be used to query the user on words that 
are not confidently recognized. The teaching system may include a speech playback system containing speech data 
ss representing prerecorded speech. This data can supply proper pronunciation of words as part of the query, thereby 
seeking user verification of a potentially misunderstood word, while at the same time pronouncing the word correctly for 
the user to hear. 

[0010] While the speech recognition apparatus of the invention is highly useful in language teaching systems, the 
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rapid adaptation system coupled with the confidence measure renders the recognizer quite useful in other applications 
where adaptation is difficult. These applications include telephone call routing and speech-enabled marketing systems 
where rapid and reliable speaker adaptation is needed almost from the instant the speaker begins speaking, 
[0011] For a more complete understanding of the invention, its objects and advantages, refer to the following spec- 
5 ificaUon and to the accompanying drawings 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0012] 

10 

Figure 1 is a block diagram of the speech recognition apparatus; 

Figure 2 is a block diagram illustrating how the confidence measure is generated using a likelihood score ratio; 
Figure 3 is a dataflow diagram illustrating how to construct an eigenspace from a plurality of training speakers; and 
Figure 4 is a data flow diagram illustrating how an adapted mode! is constructed using the eigenvoices in accord- 
is ance with the invention. 

DESCRIPTION OF THE PREFERRED EMBODIMENT 

[0013] The speech recognition apparatus of the invention is illustrated in its presently preferred form in Figure 1 . 
20 The system rapidly adapts as a new speaker 10 interacts with the recognition apparatus through the dialogue system 
12. The. dialogue system 12 can be a simple interface to speech recognizer 14, or it may include the interactive capa- 
bility to prompt the new speaker 1or input, or to provide the speaker with instruction on how to correctly pronounce words 
in the language. 

[001 4] The speech recognizer 14 operates in conjunction with one or more speech models that represent all sound 
25 units reoognized by the system. In the illustrated embodiment, an initial speech model 1 6 is associated with speech rec- 
ognizer 14 to define the speaker independent model with which the new speaker will interact during adaptation. The 
speech recognition apparatus includes an adaptation system 18 that adapts the initial speech model 16, based on 
selected, high-confidence, utterances from the new speaker. The adaptation system thus adapts the initial speech 
modei to create an adapted speech model 20 that is associated with the recognizer 14. The adapted speech model can 
so be stored apart from the initial speech model, leaving the initial speech model intact for subsequent use by different new 
speakers. Alternatively the initial speech model and adapted speech model can be merged into a single model, if 
desired. 

[0015] The adaptation system 18 performs high speed adaptation thanks to it's a priori knowledge about the class 
of new speakers. This knowledge is represented as a set of eigenspace basis vectors that we call eigenvoices. The 

35 eigenvoices are derived from a plurality of training speakers 22 and represent speaker space. The preferred embodi- 
ment represents speaker space as eigenvoice data 24. The adaptation system thus includes an eigenvoice data store 
24 containing and eigenspace data structure that represents the plurality of training speakers as a set of dimensionally 
reduced models. As will be more fully described below, the training speakers 22 provide speech upon which models are 
trained, and those models are then dimensionally reduced to generate a set of basis vectors that define the eigenspace. 

<o In essence, the eigenvoice data store represents the entire collection of training speakers. The adaptation system uses 
the basis vectors stored within data store 24 to constrain the adapted speech model 20. so that it lies within the prede- 
fined eigenspace. 

[001 6] One important advantage of the eigenvoice adaptation system is speed. The adaptation process can com- 
mence based upon a single utterance. It is not necessary for the new speaker to provide examples of each sound unit 

45 within the speech model before adaptation can commence. This advantage results from the eigenvoice representation, 
because a single utterance may be used to generate a set of vectors that may then be placed or projected onto the 
eigenspace. Although the adapted speech model 20 typically tends to improve over time, as the system is used, the 
very first adapted speech model generated by the system nevertheless tends to be remarkably accurate. 
[0017J When adapting the speech of difficult speakers, such as children or foreign speakers, some utterances may 

so be so far from the expected that they must be discarded. For example, if the dialogue system 12 prompts the new 
speaker to say the word "cat." but the speaker says the word "elephant, * that utterance should not be used to train the 
speech model associated with the word "car. While this extreme example has been provided to illustrate the point, we 
have found that young children have a great deal of difficulty concentrating on lengthy prompt-reply sessions and may 
often enter spurious utterances into the system. Young children also tend to be less aware of complying with micro- 

ss phone placement instructions. They do not always speak into the microphone at a predetermined distance. Background 
noise generated by the television r by other children can get picked up by the microphone and render certain utter- 
ances unrecognizable. ' 

[0018] To address this problem, the speech recognition apparatus uses a confidence measurement system 26 that 
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is coupled to the recognizer 14 and provides a quantitative measure of how reliable each utterance is. The confidence 
measurement system mediates the adaptation process, screening out utterances that are not reliable enough for use 
by the adaptation system 18. Although there are a number of different techniques for measuring confidence and 
employing these measurements in the adaptation system, the presently preferred embodiment uses a likelihood score 
$ ratio that compares the likelihood score associated with correct recognition with the mean or average of the likelihood 
scores associated with incorrect recognition. Further details of this likelihood ratio are provided in connection with Fig- 
ure 2 that will be discussed n ext. 

[001 9] If desired, the confidence measurement system may be coupled to the dialogue system as one mechanism 
for mediating the adaptation process. As the new speaker supplies utterances to the dialogue system, the speech rec- 
w ogmzer 14 performs speech recognition and the confidence measurement system 26 assigns a confidence measure to 
the results of that recognition. Recognized utterances with a sufficiently high confidence measure (those above a pre- 
determined confidence measure threshold) are passed by the dialogue system 1 2 to adaptation system 1 8. Utterances 
having a low confidence measure are not passed lo the adaptation system. 

[0020] Although utterances with low confidence can be simply discarded, the dialogue system 1 2 may exploit the 
is confidence measure to prompt the new speaker in a way that: (a) queries the new speaker to repeat the utterance and 
(b) teaches the new speaker the proper pronunciation for the word the speech recognizer thinks the speaker has 
uttered. Consider the following example; 

System: 'What color is the giraffe?" 
20 Child: (Answer unintelligible - low confidence, resembling "brown") 
System: "Did you say brown'?" 
Child: "Yes." 

System: Say, 'Vie giraffe is brown." 
Child: "The giraffe is brown." 

25 

In the preceding sequence, the child's initially unintelligible answer . interpreted as the word "brown" - was used in a 
subsequent series of prompts designed to teach the child the correct pronunciation. 

[0021 ] The confidence measurement system 26 can take many forms. The presently preferred embodiment uses 
the likelihood scores generated by the speech recognizer in a way that normalizes the scores to yield a measurement 
so system that is vocabulary independent Figure 2 illustrates a presently preferred embodiment of such a normalizing sys- 
tem. Input speech 40 is supplied to the speech recognizer 14 which has associated with it a plurality of speech models 
in this case Hidden K/arkoff Models 42 (HMMs). Each HMM has been trained to represent a different sound unit For 
Hlustration purposes, the HMMs shown in Figure 2 each correspond to different words that the system will recognize 
« T^Jf hnic ? es "uswted n ere are not limited to word-based models, as they are generally applicable to recognizers 
» based on other sound units (e.g., full names, syllables, words, phrases, etc.). 

[0022] In response to an input speech utterance 40. the recognizer assesses what is the likelihood that each of its 
models would generate the input utterance. The recognizer assigns or associates a recognition score with each model 
Thus, rf the input speech comprises the word "Bob", the "bob" model will generate a higher recognition score than the 
other models. 'cob\ "dog", and so forth. Illustrated at 44 are the recognition scores corresponding to the case where 
the utterance (Bob) is correctly recognized. Note that the recognition score (0.9) generated by the HMM model for Bob 
rs higher than the recognition scores generated by the other models. 
[0023] On the other hand, the input speech may be incorrectly recognized, because the utterance is mispro- 
nounced. The likelihood scores no longer identify the input utterance as the most likely, as illustrated at 40. m general 
there may be numerous examples of incorrect recognition, each with a different set of likelihood scores. This has been 
45 illustrated at 46 as a plurality of 'pages" each containing a different likelihood distribution. 

[0024] The likelihood data illustrated at 44 and 46 thus represent two different classes. The first class corresponds 
to he case of correct recognition (44) and the second class corresponds to the case of incorrect recognition (46). A ratio 
of these two classes may be constructed as depicted at 48. The ratio comprises the likelihood score for correct recog- 
cn mnoc, 5 numerator) and the mean of likelihood scores for incorrect recognition (as denominator). 

k 025] The ratio described in Figure 2 can be computed as a logarithm, in which case the log likelihood difference 
becomes a representation of whether the utterance has a high confidence or low confidence measure 
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Oj is a speech observation associated with model Sy ; 
Sj is an HMM model; 
s L(Oj\$j) is the likelihood of observation Oj given HMM model s y - ; 
y is a tuning factor. 

[0026] The above equation demonstrates that when the first term dominates (when the log likelihood difference is 
io positive) a high confidence measure results. When the second term dominates (the log likelihood difference is negative) 
a low confidence measure results. Thus, the confidence measurement system 26 (Figure 1) can readily determine 
whether a given utterance should or should not be used by the adaptation system, simply by computing the log likeli- 
hood difference according to the above equation and testing whether that difference is positive or negative or comparing 
it to a predetermined threshold. 
75 [0027] The confidence measure based on the log likelihood difference can be fine tuned by adjusting the value of 
the constant y. If the value is set very large, the sum in the second term is dominated by the largest likelihood value. 
This implies that only the contribution of the first competing word model to that of the correct recognition case is con- 
sidered. As y is made smaller, the contribution of the other likelihood's becomes more significant. In practice, y should 
be set to a value that results in an overall good verification performance for all words. 

20 

Constructing the ft penvpice Space 

[0028] The process for constructing an eigenspace to represent a plurality of training speakers is illustrated in Fig- 
ure 3. The illustration assumes a number T of training speakers 120 provide a corpus of training data 122 upon which 
2$ the eigenspace will be constructed. Preferably, a reasonably large number of speakers (on the order of 1 00 to 200) pro- 
vide the training data. These training data are then used to train a speaker dependent (SD) model as illustrated at 1 24. 
One model per speaker is constructed at step 124, with each model representing the entire lexicon that is to be under- 
stood by the recognition system. Each model can be a set of HMMs. one HMM for each sound unit. This is illustrated 
in Figure 3 at 126. 

30 [0029] After all training data from T speakers have been used to train the respective speaker dependent models, a 
set of T supervectors is constructed at 128. Thus there will be one supervector 130 for each of the T speakers. The 
supervector for each speaker comprises an ordered list of parameters (typically floating point numbers) corresponding 
to at least a portion of the parameters of the Hidden Markov Models for that speaker. Parameters corresponding to each 
sound unit are included in the supervector for a given speaker. The parameters may be organized in any convenient 

35 order. The order is not critical; however, once an order is adopted it must be followed for all T speakers. 

[0030] The ordered Hidden Markov Model parameters are then concatenated to form the supervector. The choice 
of which HMM parameters to include in the supervector may depend on the available processing power. We have found 
that constructing supervectors from the Gaussian means gives good results. If greater processing power is available, 
the supervectors may also include other HMM parameters, such as the transition probabilities or the Covariance matrix 

40 parameters. Naturally, if the Hidden Markov Models generate discrete outputs (as opposed to probability densities) then 
these output values may be used to comprise the supervector. 

[0031] After supervectors have been constructed for each of the training speakers, dimensionality reduction is per- 
formed at step 132. In general, eigenvoice adaptation involves an advantageous dimensionality reduction that can 
greatly improve the speed and efficiency at which speaker and environment adaptation is performed. Dimensionality 
<is reduction refers to a mapping of high-dimensional space onto low-dimensional space. A variety of different techniques 
may be used to effect dimensionality reduction. These indude Principal Component Analysis (PCA). Linear Discrimi- 
nant Analysis (LDA). Factor Analysis (FA). Singular Value Decomposition (SVD) and other transformations that apply 
reduction criteria based on variance. In the present example illustrated here, principal component analysis has been 
used. 

so [0032] Principal component analysis upon T supervectors yields T eigenvectors, as at 134. Thus, if 120 training 
speakers have been used the system will generate 120 eigenvectors. These eigenvectors define what we call eigen- 
voice space or eigenspace. 

[0033] The eigenvectors that make up the eigenvoice space are uncorrelated: they each represent a different 
dimension across which different speakers may be differentiated. Each supervector in the original traini ng set can be 
55 represented as a linear combination of these eigenvectors. The eigenvectors are ordered by their importance in mode- 
ling the data: the first eigenvector is more important than the second, which is more important than the third, and so on. 
Our experiments with this technique thus far show that the first eigenvector appears to correspond to a male-female 
dimension. 
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[0034] Although a maximum of T eigenvectors is produced at step 132. in practice, it is possible to discard several 
of these eigenvectors, Keeping only the first N eigenvectors. Thus at step 136 we optionally extract N of the T eigenvec- 
tors to comprise a reduced parameter eigenspace at 138. The higher order eigenvectors can be discarded because 
they typically contain less important information with which to discriminate among speakers. Reducing the eigenvoice 
space to fewer than the total number of training speakers provides an inherent data compression that can be helpful 
when constructing practical systems with limited memory and processor resources. 

Performing the Adaptation 



[0035] Once the eigenvoice space has been constructed, speaker adaptation or environment adaptation can be 
readily accomplished. While constructing the eigenspace is somewhat computationally intensive, and is typically con- 
ducted offline, adaptation is a comparatively simple computational operation that can be performed while the new 
speaker is using the system. Referring to Figure 4, speech from new speaker 140 is used at step 142 to train a speaker 
dependent model to construct a set of HMMs 144 (one for each sound unit). The speaker dependent model can be 
is trained in either a supervised mode, where the training system knows the content of the training speech in advance, or 
in an unsupervised mode, where the speech recognition system uses a speaker independent model to determine the 
content of the adaptation speech. 

[O036J The speaker dependent model trained on this new speaker will usually be very inadequate tor recognition, 
at least at first. However, the model may nevertheless be used to construct a supervector. The supervector is con- 

20 structed at step 146 such that the supervector (supervector 148) is constrained to fall within the eigenvoice space 138 
previously created from the training speakers. Supervector 148 is constructed with the imposed constraint that the 
HMM model used for recognition must be a linear combination of the eigenvoices comprising eigenvoice space iZB. 
[0037] The speaker dependent model 144 serves to estimate the linear combination of coefficients that will com- 
prise the adapted model for that new speaker. Thus in step 1 50 a new set of HMMs is constructed based on supervector 

25 148 to generate the adapted model 1 52. If desired, an optional iterative process may be performed at 1 54 to construct 
a new supervector from the adapted model 152 and thereafter to construct another set of HMMs from which a further 
adapted model maybe constructed 

[0038] The constructing of supervector 148 may be accomplished through a computationally simple projection 
operation or the like, whereby the parameters from the speaker dependent model 144 are projected or otherwise trans- 
formed or ''placed" into eigenvoice space 138. The precise technique for placing the parameters into eigenspace may 
depend upon the available computational resources. If resources are limited a simple projection operation maybe used; 
if greater resources are available an optimizing operation may be used to find the point in eigenspace that represents 
the highest probability or best match with the adaptation data. In this way, the speaker dependent model 144. trained 
on adaptation data, is combined with the eigenvoice vectors to obtain a reasonable estimate of the speaker dependent 
model for the new speaker (the adapted model 152) through a computationally inexpensive operation. 
[0039] When the new set of HMMs is constructed at step 150, any parameters that were not used in constructing 
the supervector 148 (if only Gaussian mean vectors were used for example) then the other HMM parameters may be 
obtained from a previously constructed speaker independent model. 

[0040] In some instances the adaptation data may have missing sound units (certain sound unite were not spoken 
by the new speaker). Thus the speaker dependent model 144 may not be complete. However, even with missing sound, 
units the eigenvoice adaptation technique will work At first blush this may seem surprising, however recall that the 
e.genvectors are uncorrected vectors, with the first eigenvector being more important than the second, and so forth As 
a result of this, a is possible to derive a significant amount of useful information from the adaptation data even if some 
of it is missing. 

[0041] One way to handle missing sound units is to replace the missing sound unit HMM parameters with corre- 
sponding values obtained from a speaker independent model. This is a computationally inexpensive approach, but has 
the drawback that the adapted mode! may be quite similar to the speaker independent model when only small amounts 
of adaptation data are available. 

(0042] ^ Another approach is to find The point Y in the eigenspace that maximizes the likelihood of the adaptation 
50 date, given the adapted model derived from Y. This maximum-likelihood estimator requires that we solve a different set 
of linear equations during runtime. 

[0043] From the foregoing it will be recognized that the techniques employed by the illustrated adaptation system 
can be employed m a variety of different speech recognition systems and language teaching systems. Moreover, while 
the eigenvo.ce techmques described herein are presently preferred, other techniques may be used to represent 
ss speaker space in the adaptation system. 

[0044] Accordingly, while the invention has been described in its presently prefered embodiments, it will be appre- 
ciated that the invention is capable of modification and change without departing from the spirit of the invention as set 
forth in the appended claims. 
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Claims 

1 . A speech recognition apparatus that adapts an initial speech model based on input speech from the user, compris- 
ing: 

5 

a speech model that represents speech as a plurality of speech, unit models associated with a plurality of 
speech units; 

a speech recognizer that processes input speech from a user using said speech model to recognize uttered 
speech units within said input speech; 
10 a confidence measurement system associated with said speech recognizer for associating a confidence meas- 

ure wiih each of said uttered speech units; 

an adaptation system having data store containing information reflecting a priori Knowledge about a speaker 
space; 

said adaptation system being operative to select uttered speech units that exceed a predetermined confidence 
T5 measure and to use said selected uttered speech units and said information reflecting a priori knowledge to 

adapt said speech model. 

2. The speech recognition apparatus of daim 1 wherein said adaptation system includes a data store containing a set 
of eigenspace basis vectors representing a plurality of training speakers and wherein said adaptation system uses 

20 said selected uttered speech units to train ah adapted speech model while using said basis vectors to constrain 
said adapted speech model such that said adapted speech model lies within said eigenspace. 

3. The speech recognition apparatus of claim 1 wherein said adaptation system includes a data store containing an 
eigenspace data structure that represents a plurality of training speakers as a set of models for said training speak- 

25 ers that has been dimensionally reduced to generate a set of basis vectors that define said eigenspace; and 

wherein said adaptation system uses said selected uttered speech units to train an adapted speech model 
while using said basis vectors to constrain said adapted speech mode! such that said adapted speech model lies 
within said eigenspace. 

30 4. The speech recognition apparatus of claim 1 further comprising a dialogue system coupled to said confidence 
measurement system for selecting at least a portion of said uttered speech units and for prompting said user based 
on said selected portion of said uttered speech units. 

5. The speech recognition apparatus of claim 4 further comprising speech playback system containing speech data 
3S representing prerecorded speech, said playback system coupled with said dialogue system for confirming said por- 
tion of said uttered speech units to said user by using said speech data to provide an audibl e playback correspond- 
ing to said portion of said uttered speech units. 
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FIGURE 3 
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FIGURE 4 
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